Why has (reasonably accurate) Automatic Speech Recognition been so hard to achieve?

It has now been over 35 years since hidden Markov Models were first applied to the problem of speech recognition ([2], [7]). Moreover, it has now been over 20 years since the speech recognition community seemed to collectively adopt the HMM paradigm as the most useful general approach to the fundamental problem of modeling speech. Perhaps a key turning point in this regard was Kai-Fu Lee’s thesis work [8], in which he clearly explained how to train an HMM-based system and then successfully applied a series of variations on the HMM theme to the Resource Management task, which was defined by DARPA and where the results were publicly evaluated by NIST. This is not to say that there have not been critiques of the HMM as a model of speech, nor that there have not been alternatives proposed and even explored at some length. One thinks of segmental models of various sorts ([17], [12]), and more recently, of the use of graphical models ([3]). Nonetheless, we think it is fair to say that it is still true in 2010 that the HMM remains the consensus model of choice for speech recognition, and that it lies at the heart of both commercially available products and contemporary research systems. However, in spite of the great success of the HMM paradigm, all is not well in the Land of Speech Recognition. Machine error rates on natural speech (e.g. conversational material found in Switchboard or Fisher data) are still very high (around 15% [4]), compared to what is achievable by